Skip to content

Add node-problem-detector-app, add machine health check schema #1546

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Oct 28, 2025
Merged

Add node-problem-detector-app, add machine health check schema #1546

merged 7 commits into from
Oct 28, 2025

Conversation

@AndiDog
Copy link
Contributor

@AndiDog AndiDog commented Oct 23, 2025

What this PR does / why we need it

Towards https://github.com/giantswarm/giantswarm/issues/28006.

Please read the cluster chart PR giantswarm/cluster#687, then this one should be easy.

Checklist

  • Updated CHANGELOG.md.

Trigger E2E tests

/run cluster-test-suites TARGET_SUITES=./providers/capa/standard

@AndiDog AndiDog requested a review from a team as a code owner October 23, 2025 18:39
@AndiDog AndiDog moved this to Blocked / Waiting ⛔️ in Roadmap Oct 23, 2025
@AndiDog AndiDog added this to Roadmap Oct 23, 2025
AndiDog
AndiDog commented Oct 23, 2025
View reviewed changes
@tinkerers-ci
Copy link

tinkerers-ci bot commented Oct 23, 2025

cluster-test-suites

Run name pr-cluster-aws-1546-cluster-test-suitesm8dzf
Commit SHA a708847
Result Succeeded ✅

✅ Passed test suites

CAPA Standard Suite ✅

Test Name Status Duration
BeforeSuite ✅ 10m8s
It all HelmReleases are deployed without issues ✅ 1m0s
It all default apps are deployed without issues ✅ 2m29s
It all observability-bundle apps are deployed without issues ✅ 1s
It all security-bundle apps are deployed without issues ✅ 0s
It should be able to connect to the management cluster ✅ 0s
It should be able to connect to the workload cluster ✅ 0s
It has all the control-plane nodes running ✅ 30s
It has all the worker nodes running ✅ 1m5s
It has all its Deployments Ready (means all replicas are running) ✅ 2m24s
It has all its StatefulSets Ready (means all replicas are running) ✅ 11s
It has all its DaemonSets Ready (means all daemon pods are running) ✅ 11s
It has all its Jobs completed successfully ✅ 11s
It has all of its Pods in the Running state ✅ 11s
It doesn't have restarting pods ✅ 55s
It has Cluster Ready condition with Status='True' ✅ 0s
It has all machine pools ready and running ✅ 30s
It cert-manager default ClusterIssuers are present and ready ✅ 0s
It sets up the api DNS records ✅ 0s
It sets up the bastion DNS records ⏭️ 0s
It should have cert-manager and external-dns deployed ✅ 0s
It should deploy ingress-nginx ✅ 16s
It cluster wildcard ingress DNS must be resolvable ✅ 1m1s
It should deploy the hello-world app ✅ 6s
It ingress resource has load balancer in status ✅ 50s
It should have a ready Certificate generated ✅ 0s
It hello world app responds successfully ✅ 0s
It uninstall apps ✅ 1s
It creates test pod ✅ 5s
It ensure key metrics are available on mimir ✅ 8s
It clean up test pod ✅ 35s
It scales node by creating anti-affinity pods ✅ 1m18s
It has a at least one storage class available ✅ 11s
It creates the new namespace for the test ✅ 0s
It creates the PVC ✅ 0s
It creates the pod using the PVC ✅ 0s
It binds the PVC ✅ 10s
It runs successfully ✅ 11s
It deletes all resources correct ✅ 20s
It cluster is registered ✅ 0s
It should be able to pull an image from a private ECR registry ✅ 10s
AfterSuite ✅ 11m45s

📋 View full results in Tekton Dashboard

Rerun trigger:
/run cluster-test-suites

Tip

To only re-run the failed test suites you can provide a TARGET_SUITES parameter with your trigger that points to the directory path of the test suites to run, e.g. /run cluster-test-suites TARGET_SUITES=./providers/capa/standard to re-run the CAPA standard test suite. This supports multiple test suites with each path separated by a comma.

To run this test suite as a major upgrade, which will test upgrading from the latest release of the previous major version, you can add IS_MAJOR_UPGRADE=true, e.g. /run cluster-test-suites IS_MAJOR_UPGRADE=true.

Available Test Suites

By default, only the standard test suite runs to reduce costs. If your changes affect specialized environments, you can specify additional test suites:

AWS (CAPA) Test Suites

  • standard - Basic cluster creation and functionality
  • karpenter - Karpenter cluster creation testing
  • china - China-specific environment testing
  • private - Private cloud environment testing
  • cilium-eni-mode - Cilium ENI mode testing
  • upgrade - Cluster upgrade testing
  • upgrade-major - Major version upgrade testing

How to Specify Additional Test Suites

# Run specific test suites
/run cluster-test-suites TARGET_SUITES=./providers/capa/standard,./providers/capa/china

# Run all test suites for CAPA
/run cluster-test-suites TARGET_SUITES=./providers/capa/

# Run upgrade tests
/run cluster-test-suites TARGET_SUITES=./providers/capa/upgrade,./providers/capa/upgrade-major

Note: Full test suites run automatically on releases. You are responsible for testing all relevant flavors before merging.

@AndiDog AndiDog force-pushed the node-problem-detector branch from a708847 to 88966a8 Compare October 27, 2025 08:41
fiunchinho
fiunchinho reviewed Oct 27, 2025
View reviewed changes
helm/cluster-aws/values.schema.json
"title": "Enable",
"default": true
},
"maxUnhealthy": {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is missing a description.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

fiunchinho
fiunchinho reviewed Oct 27, 2025
View reviewed changes
@AndiDog
Copy link
Contributor Author

AndiDog commented Oct 28, 2025

/run cluster-test-suites

@github-actions
Copy link
Contributor

There were differences in the rendered Helm template, please check! ⚠️

Output 
=== Differences when rendered with values file helm/cluster-aws/ci/test-auditd-values.yaml ===

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-cert-manager-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-cert-manager-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-cluster-autoscaler-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-cluster-autoscaler-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-etcd-defrag-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-etcd-defrag-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-etcd-k8s-res-count-exporter-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-etcd-k8s-res-count-exporter-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-external-dns-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-external-dns-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-metrics-server-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-metrics-server-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-net-exporter-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-net-exporter-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-security-bundle-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-security-bundle-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cert-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cert-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cert-manager)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cert-manager)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-chart-operator-extensions)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-chart-operator-extensions)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cilium-servicemonitors)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cilium-servicemonitors)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cluster-autoscaler)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cluster-autoscaler)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-coredns-extensions)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-coredns-extensions)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-etcd-defrag)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-etcd-defrag)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-etcd-k8s-res-count-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-etcd-k8s-res-count-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-external-dns)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-external-dns)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-k8s-audit-metrics)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-k8s-audit-metrics)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-k8s-dns-node-cache)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-k8s-dns-node-cache)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-metrics-server)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-metrics-server)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-net-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-net-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-node-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-node-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-observability-bundle)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-observability-bundle)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-observability-policies)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-observability-policies)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-prometheus-blackbox-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-prometheus-blackbox-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-security-bundle)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-security-bundle)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-teleport-kube-agent)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-teleport-kube-agent)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-vertical-pod-autoscaler)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-vertical-pod-autoscaler)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/Cluster/org-giantswarm/test-wc-minimal)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/Cluster/org-giantswarm/test-wc-minimal)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-minimal-cilium)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-minimal-cilium)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-minimal-coredns)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-minimal-coredns)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-minimal-network-policies)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-minimal-network-policies)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-minimal-vertical-pod-autoscaler-crd)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-minimal-vertical-pod-autoscaler-crd)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-minimal-default)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-minimal-default)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-minimal-default-test)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-minimal-default-test)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-minimal-cluster)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-minimal-cluster)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-minimal-cluster-test)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-minimal-cluster-test)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (bootstrap.cluster.x-k8s.io/v1beta1/KubeadmConfig/org-giantswarm/test-wc-minimal-pool0-68ccf)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (bootstrap.cluster.x-k8s.io/v1beta1/KubeadmConfig/org-giantswarm/test-wc-minimal-pool0-68ccf)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-wc-minimal)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-wc-minimal)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/spec/machineTemplate/metadata/labels/app.kubernetes.io/version  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-wc-minimal)
  ± value change
    - 4.3.0
    + 4.4.0

/spec/machineTemplate/metadata/labels/helm.sh/chart  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-wc-minimal)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/MachineHealthCheck/org-giantswarm/test-wc-minimal-control-plane)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/MachineHealthCheck/org-giantswarm/test-wc-minimal-control-plane)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/MachinePool/org-giantswarm/test-wc-minimal-pool0)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/MachinePool/org-giantswarm/test-wc-minimal-pool0)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ServiceAccount/org-giantswarm/test-wc-minimal-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ServiceAccount/org-giantswarm/test-wc-minimal-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (rbac.authorization.k8s.io/v1/Role/org-giantswarm/test-wc-minimal-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (rbac.authorization.k8s.io/v1/Role/org-giantswarm/test-wc-minimal-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/rules  (rbac.authorization.k8s.io/v1/Role/org-giantswarm/test-wc-minimal-helmreleases-cleanup)
  + one list entry added:
    - resources:
      - helmcharts
      - helmcharts/finalizers
      apiGroups:
      - source.toolkit.fluxcd.io
      verbs:
      - delete
      - get
      - list
      - patch

/metadata/labels/app.kubernetes.io/version  (rbac.authorization.k8s.io/v1/RoleBinding/org-giantswarm/test-wc-minimal-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (rbac.authorization.k8s.io/v1/RoleBinding/org-giantswarm/test-wc-minimal-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (batch/v1/Job/org-giantswarm/test-wc-minimal-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (batch/v1/Job/org-giantswarm/test-wc-minimal-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/spec/template/metadata/labels/app.kubernetes.io/version  (batch/v1/Job/org-giantswarm/test-wc-minimal-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/spec/template/metadata/labels/helm.sh/chart  (batch/v1/Job/org-giantswarm/test-wc-minimal-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/spec/template/spec/containers/name=kubectl/args  (batch/v1/Job/org-giantswarm/test-wc-minimal-helmreleases-cleanup)
  - one list entry removed:
    - |
      # Print namespace & cluster.
      echo "# Namespace: ${NAMESPACE} | Cluster: ${CLUSTER}"
      
      # Get releases.
      releases="$(kubectl get helmreleases.helm.toolkit.fluxcd.io --namespace "${NAMESPACE}" --selector giantswarm.io/cluster="${CLUSTER}" --output name)"
      
      # Check releases.
      if [ -n "${releases}" ]
      then
        # Patch releases.
        kubectl patch --namespace "${NAMESPACE}" ${releases} --type merge --patch '{ "spec": { "suspend": true } }'
      else
        # Print info.
        echo "No releases to patch found."
      fi
      
    
  
  + one list entry added:
    - |
      # Print namespace & cluster.
      echo "# Namespace: ${NAMESPACE} | Cluster: ${CLUSTER}"
      
      # Get releases.
      releases="$(kubectl get helmreleases.helm.toolkit.fluxcd.io --namespace "${NAMESPACE}" --selector giantswarm.io/cluster="${CLUSTER}" --output name)"
      
      # Check releases.
      if [ -n "${releases}" ]
      then
        # Patch releases.
        kubectl patch --namespace "${NAMESPACE}" ${releases} --type merge --patch '{ "spec": { "suspend": true } }'
      
        # Get and delete associated HelmChart CRs
        # HelmChart CRs don't have the giantswarm.io/cluster label, so we need to find them by name pattern
        charts="$(kubectl get helmcharts.source.toolkit.fluxcd.io --namespace "${NAMESPACE}" --output name | grep "^helmchart.source.toolkit.fluxcd.io/${NAMESPACE}-${CLUSTER}-")"
        if [ -n "${charts}" ]
        then
          echo "Deleting HelmChart CRs: ${charts}"
          # Remove finalizers and delete HelmChart CRs
          for chart in ${charts}; do
            kubectl patch --namespace "${NAMESPACE}" ${chart} --type json --patch '[{"op": "remove", "path": "/metadata/finalizers"}]'
            kubectl delete --namespace "${NAMESPACE}" ${chart}
          done
        else
          echo "No HelmChart CRs to delete found."
        fi
      else
        # Print info.
        echo "No releases to patch found."
      fi
      
    
  



=== Differences when rendered with values file helm/cluster-aws/ci/test-eni-mode-values.yaml ===

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-cert-manager-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-cert-manager-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-cluster-autoscaler-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-cluster-autoscaler-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-etcd-defrag-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-etcd-defrag-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-etcd-k8s-res-count-exporter-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-etcd-k8s-res-count-exporter-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-external-dns-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-external-dns-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-metrics-server-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-metrics-server-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-net-exporter-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-net-exporter-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-security-bundle-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-security-bundle-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-cert-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-cert-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-cert-manager)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-cert-manager)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-chart-operator-extensions)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-chart-operator-extensions)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-cilium-servicemonitors)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-cilium-servicemonitors)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-cluster-autoscaler)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-cluster-autoscaler)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-coredns-extensions)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-coredns-extensions)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-etcd-defrag)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-etcd-defrag)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-etcd-k8s-res-count-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-etcd-k8s-res-count-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-external-dns)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-external-dns)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-k8s-audit-metrics)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-k8s-audit-metrics)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-k8s-dns-node-cache)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-k8s-dns-node-cache)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-metrics-server)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-metrics-server)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-net-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-net-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-node-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-node-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-observability-bundle)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-observability-bundle)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-observability-policies)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-observability-policies)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-prometheus-blackbox-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-prometheus-blackbox-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-security-bundle)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-security-bundle)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-teleport-kube-agent)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-teleport-kube-agent)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-vertical-pod-autoscaler)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-vertical-pod-autoscaler)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/Cluster/org-giantswarm/test-wc)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/Cluster/org-giantswarm/test-wc)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-cilium)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-cilium)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-coredns)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-coredns)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-network-policies)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-network-policies)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-vertical-pod-autoscaler-crd)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-wc-vertical-pod-autoscaler-crd)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-default)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-default)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-default-test)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-default-test)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-cluster)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-cluster)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-cluster-test)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-wc-cluster-test)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (bootstrap.cluster.x-k8s.io/v1beta1/KubeadmConfig/org-giantswarm/test-wc-pool0-b8ad0)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (bootstrap.cluster.x-k8s.io/v1beta1/KubeadmConfig/org-giantswarm/test-wc-pool0-b8ad0)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-wc)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-wc)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/spec/machineTemplate/metadata/labels/app.kubernetes.io/version  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-wc)
  ± value change
    - 4.3.0
    + 4.4.0

/spec/machineTemplate/metadata/labels/helm.sh/chart  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-wc)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/MachineHealthCheck/org-giantswarm/test-wc-control-plane)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/MachineHealthCheck/org-giantswarm/test-wc-control-plane)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/MachinePool/org-giantswarm/test-wc-pool0)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/MachinePool/org-giantswarm/test-wc-pool0)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ServiceAccount/org-giantswarm/test-wc-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ServiceAccount/org-giantswarm/test-wc-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (rbac.authorization.k8s.io/v1/Role/org-giantswarm/test-wc-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (rbac.authorization.k8s.io/v1/Role/org-giantswarm/test-wc-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/rules  (rbac.authorization.k8s.io/v1/Role/org-giantswarm/test-wc-helmreleases-cleanup)
  + one list entry added:
    - resources:
      - helmcharts
      - helmcharts/finalizers
      apiGroups:
      - source.toolkit.fluxcd.io
      verbs:
      - delete
      - get
      - list
      - patch

/metadata/labels/app.kubernetes.io/version  (rbac.authorization.k8s.io/v1/RoleBinding/org-giantswarm/test-wc-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (rbac.authorization.k8s.io/v1/RoleBinding/org-giantswarm/test-wc-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (batch/v1/Job/org-giantswarm/test-wc-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (batch/v1/Job/org-giantswarm/test-wc-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/spec/template/metadata/labels/app.kubernetes.io/version  (batch/v1/Job/org-giantswarm/test-wc-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/spec/template/metadata/labels/helm.sh/chart  (batch/v1/Job/org-giantswarm/test-wc-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/spec/template/spec/containers/name=kubectl/args  (batch/v1/Job/org-giantswarm/test-wc-helmreleases-cleanup)
  - one list entry removed:
    - |
      # Print namespace & cluster.
      echo "# Namespace: ${NAMESPACE} | Cluster: ${CLUSTER}"
      
      # Get releases.
      releases="$(kubectl get helmreleases.helm.toolkit.fluxcd.io --namespace "${NAMESPACE}" --selector giantswarm.io/cluster="${CLUSTER}" --output name)"
      
      # Check releases.
      if [ -n "${releases}" ]
      then
        # Patch releases.
        kubectl patch --namespace "${NAMESPACE}" ${releases} --type merge --patch '{ "spec": { "suspend": true } }'
      else
        # Print info.
        echo "No releases to patch found."
      fi
      
    
  
  + one list entry added:
    - |
      # Print namespace & cluster.
      echo "# Namespace: ${NAMESPACE} | Cluster: ${CLUSTER}"
      
      # Get releases.
      releases="$(kubectl get helmreleases.helm.toolkit.fluxcd.io --namespace "${NAMESPACE}" --selector giantswarm.io/cluster="${CLUSTER}" --output name)"
      
      # Check releases.
      if [ -n "${releases}" ]
      then
        # Patch releases.
        kubectl patch --namespace "${NAMESPACE}" ${releases} --type merge --patch '{ "spec": { "suspend": true } }'
      
        # Get and delete associated HelmChart CRs
        # HelmChart CRs don't have the giantswarm.io/cluster label, so we need to find them by name pattern
        charts="$(kubectl get helmcharts.source.toolkit.fluxcd.io --namespace "${NAMESPACE}" --output name | grep "^helmchart.source.toolkit.fluxcd.io/${NAMESPACE}-${CLUSTER}-")"
        if [ -n "${charts}" ]
        then
          echo "Deleting HelmChart CRs: ${charts}"
          # Remove finalizers and delete HelmChart CRs
          for chart in ${charts}; do
            kubectl patch --namespace "${NAMESPACE}" ${chart} --type json --patch '[{"op": "remove", "path": "/metadata/finalizers"}]'
            kubectl delete --namespace "${NAMESPACE}" ${chart}
          done
        else
          echo "No HelmChart CRs to delete found."
        fi
      else
        # Print info.
        echo "No releases to patch found."
      fi
      
    
  



=== Differences when rendered with values file helm/cluster-aws/ci/test-hop-count-tuning-values.yaml ===

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-cert-manager-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-cert-manager-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-cluster-autoscaler-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-cluster-autoscaler-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-etcd-defrag-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-etcd-defrag-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-etcd-k8s-res-count-exporter-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-etcd-k8s-res-count-exporter-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-external-dns-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-external-dns-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-metrics-server-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-metrics-server-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-net-exporter-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-net-exporter-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-security-bundle-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-hop-count-tuning-security-bundle-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-cert-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-cert-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-cert-manager)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-cert-manager)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-chart-operator-extensions)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-chart-operator-extensions)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-cilium-servicemonitors)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-cilium-servicemonitors)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-cluster-autoscaler)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-cluster-autoscaler)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-coredns-extensions)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-coredns-extensions)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-etcd-defrag)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-etcd-defrag)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-etcd-k8s-res-count-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-etcd-k8s-res-count-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-external-dns)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-external-dns)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-k8s-audit-metrics)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-k8s-audit-metrics)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-k8s-dns-node-cache)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-k8s-dns-node-cache)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-metrics-server)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-metrics-server)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-net-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-net-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-node-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-node-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-observability-bundle)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-observability-bundle)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-observability-policies)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-observability-policies)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-prometheus-blackbox-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-prometheus-blackbox-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-security-bundle)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-security-bundle)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-teleport-kube-agent)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-teleport-kube-agent)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-vertical-pod-autoscaler)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-hop-count-tuning-vertical-pod-autoscaler)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/Cluster/org-giantswarm/test-hop-count-tuning)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/Cluster/org-giantswarm/test-hop-count-tuning)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-hop-count-tuning-cilium)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-hop-count-tuning-cilium)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-hop-count-tuning-coredns)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-hop-count-tuning-coredns)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-hop-count-tuning-network-policies)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-hop-count-tuning-network-policies)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-hop-count-tuning-vertical-pod-autoscaler-crd)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (helm.toolkit.fluxcd.io/v2/HelmRelease/org-giantswarm/test-hop-count-tuning-vertical-pod-autoscaler-crd)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-hop-count-tuning-default)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-hop-count-tuning-default)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-hop-count-tuning-default-test)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-hop-count-tuning-default-test)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-hop-count-tuning-cluster)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-hop-count-tuning-cluster)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-hop-count-tuning-cluster-test)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (source.toolkit.fluxcd.io/v1/HelmRepository/org-giantswarm/test-hop-count-tuning-cluster-test)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (bootstrap.cluster.x-k8s.io/v1beta1/KubeadmConfig/org-giantswarm/test-hop-count-tuning-pool0-8c673)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (bootstrap.cluster.x-k8s.io/v1beta1/KubeadmConfig/org-giantswarm/test-hop-count-tuning-pool0-8c673)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-hop-count-tuning)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-hop-count-tuning)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/spec/machineTemplate/metadata/labels/app.kubernetes.io/version  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-hop-count-tuning)
  ± value change
    - 4.3.0
    + 4.4.0

/spec/machineTemplate/metadata/labels/helm.sh/chart  (controlplane.cluster.x-k8s.io/v1beta1/KubeadmControlPlane/org-giantswarm/test-hop-count-tuning)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/MachineHealthCheck/org-giantswarm/test-hop-count-tuning-control-plane)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/MachineHealthCheck/org-giantswarm/test-hop-count-tuning-control-plane)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (cluster.x-k8s.io/v1beta1/MachinePool/org-giantswarm/test-hop-count-tuning-pool0)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (cluster.x-k8s.io/v1beta1/MachinePool/org-giantswarm/test-hop-count-tuning-pool0)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ServiceAccount/org-giantswarm/test-hop-count-tuning-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ServiceAccount/org-giantswarm/test-hop-count-tuning-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (rbac.authorization.k8s.io/v1/Role/org-giantswarm/test-hop-count-tuning-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (rbac.authorization.k8s.io/v1/Role/org-giantswarm/test-hop-count-tuning-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/rules  (rbac.authorization.k8s.io/v1/Role/org-giantswarm/test-hop-count-tuning-helmreleases-cleanup)
  + one list entry added:
    - resources:
      - helmcharts
      - helmcharts/finalizers
      apiGroups:
      - source.toolkit.fluxcd.io
      verbs:
      - delete
      - get
      - list
      - patch

/metadata/labels/app.kubernetes.io/version  (rbac.authorization.k8s.io/v1/RoleBinding/org-giantswarm/test-hop-count-tuning-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (rbac.authorization.k8s.io/v1/RoleBinding/org-giantswarm/test-hop-count-tuning-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (batch/v1/Job/org-giantswarm/test-hop-count-tuning-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (batch/v1/Job/org-giantswarm/test-hop-count-tuning-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/spec/template/metadata/labels/app.kubernetes.io/version  (batch/v1/Job/org-giantswarm/test-hop-count-tuning-helmreleases-cleanup)
  ± value change
    - 4.3.0
    + 4.4.0

/spec/template/metadata/labels/helm.sh/chart  (batch/v1/Job/org-giantswarm/test-hop-count-tuning-helmreleases-cleanup)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/spec/template/spec/containers/name=kubectl/args  (batch/v1/Job/org-giantswarm/test-hop-count-tuning-helmreleases-cleanup)
  - one list entry removed:
    - |
      # Print namespace & cluster.
      echo "# Namespace: ${NAMESPACE} | Cluster: ${CLUSTER}"
      
      # Get releases.
      releases="$(kubectl get helmreleases.helm.toolkit.fluxcd.io --namespace "${NAMESPACE}" --selector giantswarm.io/cluster="${CLUSTER}" --output name)"
      
      # Check releases.
      if [ -n "${releases}" ]
      then
        # Patch releases.
        kubectl patch --namespace "${NAMESPACE}" ${releases} --type merge --patch '{ "spec": { "suspend": true } }'
      else
        # Print info.
        echo "No releases to patch found."
      fi
      
    
  
  + one list entry added:
    - |
      # Print namespace & cluster.
      echo "# Namespace: ${NAMESPACE} | Cluster: ${CLUSTER}"
      
      # Get releases.
      releases="$(kubectl get helmreleases.helm.toolkit.fluxcd.io --namespace "${NAMESPACE}" --selector giantswarm.io/cluster="${CLUSTER}" --output name)"
      
      # Check releases.
      if [ -n "${releases}" ]
      then
        # Patch releases.
        kubectl patch --namespace "${NAMESPACE}" ${releases} --type merge --patch '{ "spec": { "suspend": true } }'
      
        # Get and delete associated HelmChart CRs
        # HelmChart CRs don't have the giantswarm.io/cluster label, so we need to find them by name pattern
        charts="$(kubectl get helmcharts.source.toolkit.fluxcd.io --namespace "${NAMESPACE}" --output name | grep "^helmchart.source.toolkit.fluxcd.io/${NAMESPACE}-${CLUSTER}-")"
        if [ -n "${charts}" ]
        then
          echo "Deleting HelmChart CRs: ${charts}"
          # Remove finalizers and delete HelmChart CRs
          for chart in ${charts}; do
            kubectl patch --namespace "${NAMESPACE}" ${chart} --type json --patch '[{"op": "remove", "path": "/metadata/finalizers"}]'
            kubectl delete --namespace "${NAMESPACE}" ${chart}
          done
        else
          echo "No HelmChart CRs to delete found."
        fi
      else
        # Print info.
        echo "No releases to patch found."
      fi
      
    
  



=== Differences when rendered with values file helm/cluster-aws/ci/test-irsa-crossplane-values.yaml ===

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-cert-manager-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-cert-manager-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-cluster-autoscaler-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-cluster-autoscaler-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-etcd-defrag-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-etcd-defrag-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-etcd-k8s-res-count-exporter-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-etcd-k8s-res-count-exporter-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-external-dns-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-external-dns-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-metrics-server-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-metrics-server-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-net-exporter-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-net-exporter-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (v1/ConfigMap/org-giantswarm/test-wc-minimal-security-bundle-user-values)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (v1/ConfigMap/org-giantswarm/test-wc-minimal-security-bundle-user-values)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cert-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cert-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cert-manager)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cert-manager)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-chart-operator-extensions)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-chart-operator-extensions)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cilium-servicemonitors)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cilium-servicemonitors)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cluster-autoscaler)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-cluster-autoscaler)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-coredns-extensions)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-coredns-extensions)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-etcd-defrag)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-etcd-defrag)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-etcd-k8s-res-count-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-etcd-k8s-res-count-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-external-dns)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-external-dns)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-k8s-audit-metrics)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-k8s-audit-metrics)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-k8s-dns-node-cache)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-k8s-dns-node-cache)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-metrics-server)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-metrics-server)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-net-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-net-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-node-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-node-exporter)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-observability-bundle)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-observability-bundle)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-observability-policies)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-observability-policies)
  ± value change
    - cluster-4.3.0
    + cluster-4.4.0

/metadata/labels/app.kubernetes.io/version  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-prometheus-blackbox-exporter)
  ± value change
    - 4.3.0
    + 4.4.0

/metadata/labels/helm.sh/chart  (application.giantswarm.io/v1alpha1/App/org-giantswarm/test-wc-minimal-prometheus-blackbox-exporter)
 ...*[Comment body truncated]*

@tinkerers-ci
Copy link

tinkerers-ci bot commented Oct 28, 2025

cluster-test-suites

Run name pr-cluster-aws-1546-cluster-test-suites6rt68
Commit SHA de8701f
Result Succeeded ✅

✅ Passed test suites

CAPA Standard Suite ✅

Test Name Status Duration
BeforeSuite ✅ 11m50s
It all HelmReleases are deployed without issues ✅ 1m0s
It all default apps are deployed without issues ✅ 2m0s
It all observability-bundle apps are deployed without issues ✅ 1s
It all security-bundle apps are deployed without issues ✅ 0s
It should be able to connect to the management cluster ✅ 0s
It should be able to connect to the workload cluster ✅ 0s
It has all the control-plane nodes running ✅ 30s
It has all the worker nodes running ✅ 1m5s
It has all its Deployments Ready (means all replicas are running) ✅ 33s
It has all its StatefulSets Ready (means all replicas are running) ✅ 11s
It has all its DaemonSets Ready (means all daemon pods are running) ✅ 11s
It has all its Jobs completed successfully ✅ 11s
It has all of its Pods in the Running state ✅ 33s
It doesn't have restarting pods ✅ 55s
It has Cluster Ready condition with Status='True' ✅ 0s
It has all machine pools ready and running ✅ 30s
It cert-manager default ClusterIssuers are present and ready ✅ 0s
It sets up the api DNS records ✅ 0s
It sets up the bastion DNS records ⏭️ 0s
It should have cert-manager and external-dns deployed ✅ 0s
It should deploy ingress-nginx ✅ 11s
It cluster wildcard ingress DNS must be resolvable ✅ 52s
It should deploy the hello-world app ✅ 6s
It ingress resource has load balancer in status ✅ 5s
It should have a ready Certificate generated ✅ 20s
It hello world app responds successfully ✅ 0s
It uninstall apps ✅ 1s
It creates test pod ✅ 5s
It ensure key metrics are available on mimir ✅ 8s
It clean up test pod ✅ 35s
It scales node by creating anti-affinity pods ✅ 1m29s
It has a at least one storage class available ✅ 11s
It creates the new namespace for the test ✅ 0s
It creates the PVC ✅ 0s
It creates the pod using the PVC ✅ 0s
It binds the PVC ✅ 10s
It runs successfully ✅ 22s
It deletes all resources correct ✅ 20s
It cluster is registered ✅ 0s
It should be able to pull an image from a private ECR registry ✅ 10s
AfterSuite ✅ 16m37s

📋 View full results in Tekton Dashboard

Rerun trigger:
/run cluster-test-suites

Tip

To only re-run the failed test suites you can provide a TARGET_SUITES parameter with your trigger that points to the directory path of the test suites to run, e.g. /run cluster-test-suites TARGET_SUITES=./providers/capa/standard to re-run the CAPA standard test suite. This supports multiple test suites with each path separated by a comma.

To run this test suite as a major upgrade, which will test upgrading from the latest release of the previous major version, you can add IS_MAJOR_UPGRADE=true, e.g. /run cluster-test-suites IS_MAJOR_UPGRADE=true.

Available Test Suites

By default, only the standard test suite runs to reduce costs. If your changes affect specialized environments, you can specify additional test suites:

AWS (CAPA) Test Suites

  • standard - Basic cluster creation and functionality
  • karpenter - Karpenter cluster creation testing
  • china - China-specific environment testing
  • private - Private cloud environment testing
  • cilium-eni-mode - Cilium ENI mode testing
  • upgrade - Cluster upgrade testing
  • upgrade-major - Major version upgrade testing

How to Specify Additional Test Suites

# Run specific test suites
/run cluster-test-suites TARGET_SUITES=./providers/capa/standard,./providers/capa/china

# Run all test suites for CAPA
/run cluster-test-suites TARGET_SUITES=./providers/capa/

# Run upgrade tests
/run cluster-test-suites TARGET_SUITES=./providers/capa/upgrade,./providers/capa/upgrade-major

Note: Full test suites run automatically on releases. You are responsible for testing all relevant flavors before merging.

@AndiDog AndiDog merged commit 760cc03 into main Oct 28, 2025
12 of 13 checks passed
@AndiDog AndiDog deleted the node-problem-detector branch October 28, 2025 16:44
@github-project-automation github-project-automation bot moved this from Blocked / Waiting ⛔️ to Done ✅ in Roadmap Oct 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fiunchinho fiunchinho fiunchinho approved these changes

Assignees

No one assigned

Labels

None yet

Projects

Roadmap
Status: Done ✅

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@AndiDog @fiunchinho @Gacko